Reinforcement learning can enable robots to navigate to distant goals while optimizing user-specified reward functions, including preferences for following lanes, staying on paved paths, or avoiding freshly mowed grass. However, online learning from trial-and-error for real-world robots is logistically challenging, and methods that instead can utilize existing datasets of robotic navigation data could be significantly more scalable and enable broader generalization. In this paper, we present ReViND, the first offline RL system for robotic navigation that can leverage previously collected data to optimize user-specified reward functions in the real-world. We evaluate our system for off-road navigation without any additional data collection or fine-tuning, and show that it can navigate to distant goals using only offline training from this dataset, and exhibit behaviors that qualitatively differ based on the user-specified reward function.

深度强化学习是在不需要领域知识的不受控制环境中学习政策的有前途的方法。不幸的是，由于样本效率低下，深度RL应用主要集中在模拟环境上。在这项工作中，我们证明了机器学习算法和库的最新进步与精心调整的机器人控制器相结合，导致在现实世界中仅20分钟内学习四倍的运动。我们在几个室内和室外地形上评估了我们的方法，这些室内和室外地形对基于古典模型的控制器来说是具有挑战性的。我们观察机器人能够在所有这些地形上始终如一地学习步态。最后，我们在模拟环境中评估我们的设计决策。

最近的多任务学习研究旨在反对单一的标准化，其中培训只需最大限度地减少任务损失的总和。代替了几种Ad-hoc多任务优化算法，它受到各种假设的启发，关于使多任务设置困难的原因。这些优化器中的大多数都需要每个任务渐变，并引入重要的内存，运行时和实现开销。我们提出了一个理论分析，表明许多专业的多任务优化器可以被解释为正规化的形式。此外，我们表明，当与单任务学习的标准正则化和稳定技术耦合时，单一的标定化匹配或改善在监督和加固学习设置中复杂的多任务优化器的性能。我们相信我们的结果要求对该地区最近的研究进行关键重新评估。

最近的工作表明，单独监督学习，没有时间差异（TD）学习，可以对离线RL显着有效。什么时候保持真实，需要哪些算法组件？通过广泛的实验，我们致力于将RL离线的监督学习到其基本要素。在我们考虑的每个环境套件中，只需通过双层前馈MLP最大化的可能性，与基于TD学习或与变压器的序列建模的基本更复杂的方法具有竞争力的竞争性。仔细选择模型容量（例如，通过正则化或架构），并选择哪些信息（例如，目标或奖励）对性能至关重要。这些见解是通过监督学习进行加强学习的从业者（我们投入“RVS学习”）的实践指南。他们还探讨了现有RVS方法的限制，在随机数据上相对较弱，并提出了许多打开问题。

强化学习（RL）代理商广泛用于解决复杂的连续决策任务，但仍然表现出概括到培训期间未见的情景。在先前的在线方法证明，使用超出奖励功能的其他信号可以导致RL代理商中的更好的泛化能力，即使用自我监督学习（SSL），他们在离线RL设置中奋斗，即从静态数据集中学习。我们表明，由于观察之间的相似性差异差，可以在离线设置中阻碍用于RL的普遍的在线算法的性能。我们提出了一种称为广义相似性功能（GSF）的新的理论上动机框架，它使用对比学习来训练基于其预期未来行为的相似性的离线RL代理，以便使用\ EMPH {广义值来量化此相似性。职能}。我们表明GSF足以恢复现有的SSL目标，同时还可以在复杂的离线RL基准，离线Procgen上提高零拍泛化性能。

We propose a simple data augmentation technique that can be applied to standard model-free reinforcement learning algorithms, enabling robust learning directly from pixels without the need for auxiliary losses or pre-training. The approach leverages input perturbations commonly used in computer vision tasks to transform input examples, as well as regularizing the value function and policy. Existing model-free approaches, such as Soft Actor-Critic (SAC) [22], are not able to train deep networks effectively from image pixels. However, the addition of our augmentation method dramatically improves SAC's performance, enabling it to reach state-of-the-art performance on the DeepMind control suite, surpassing model-based [23,38,24] methods and recently proposed contrastive learning [50]. Our approach, which we dub DrQ: Data-regularized Q, can be combined with any model-free reinforcement learning algorithm. We further demonstrate this by applying it to DQN [43] and significantly improve its data-efficiency on the Atari 100k [31] benchmark. An implementation can be found at https://sites. google.com/view/data-regularized-q.

We present a dynamic path planning algorithm to navigate an amphibious rotor craft through a concave time-invariant obstacle field while attempting to minimize energy usage. We create a nonlinear quaternion state model that represents the rotor craft dynamics above and below the water. The 6 degree of freedom dynamics used within a layered architecture to generate motion paths for the vehicle to follow and the required control inputs. The rotor craft has a 3 dimensional map of its surroundings that is updated via limited range onboard sensor readings within the current medium (air or water). Path planning is done via PRM and D* Lite.

Biological systems often choose actions without an explicit reward signal, a phenomenon known as intrinsic motivation. The computational principles underlying this behavior remain poorly understood. In this study, we investigate an information-theoretic approach to intrinsic motivation, based on maximizing an agent's empowerment (the mutual information between its past actions and future states). We show that this approach generalizes previous attempts to formalize intrinsic motivation, and we provide a computationally efficient algorithm for computing the necessary quantities. We test our approach on several benchmark control problems, and we explain its success in guiding intrinsically motivated behaviors by relating our information-theoretic control function to fundamental properties of the dynamical system representing the combined agent-environment system. This opens the door for designing practical artificial, intrinsically motivated controllers and for linking animal behaviors to their dynamical properties.

Modern mobile burst photography pipelines capture and merge a short sequence of frames to recover an enhanced image, but often disregard the 3D nature of the scene they capture, treating pixel motion between images as a 2D aggregation problem. We show that in a "long-burst", forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. To this end, we devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion. Our plane plus depth model is trained end-to-end, and performs coarse-to-fine refinement by controlling which multi-resolution volume features the network has access to at what time during training. We validate the method experimentally, and demonstrate geometrically accurate depth reconstructions with no additional hardware or separate data pre-processing and pose-estimation steps.

Recent advances in batch (offline) reinforcement learning have shown promising results in learning from available offline data and proved offline reinforcement learning to be an essential toolkit in learning control policies in a model-free setting. An offline reinforcement learning algorithm applied to a dataset collected by a suboptimal non-learning-based algorithm can result in a policy that outperforms the behavior agent used to collect the data. Such a scenario is frequent in robotics, where existing automation is collecting operational data. Although offline learning techniques can learn from data generated by a sub-optimal behavior agent, there is still an opportunity to improve the sample complexity of existing offline reinforcement learning algorithms by strategically introducing human demonstration data into the training process. To this end, we propose a novel approach that uses uncertainty estimation to trigger the injection of human demonstration data and guide policy training towards optimal behavior while reducing overall sample complexity. Our experiments show that this approach is more sample efficient when compared to a naive way of combining expert data with data collected from a sub-optimal agent. We augmented an existing offline reinforcement learning algorithm Conservative Q-Learning with our approach and performed experiments on data collected from MuJoCo and OffWorld Gym learning environments.